System and method for establishing and managing multiple call sessions from a centralized control interface

ABSTRACT

Disclosed are a system and method for establishing and managing one-to-one and conference call sessions through a virtual waiting room. Conference calls may be established initially or created as additional people are invited to an existing call. Functions such as screensharing, chat messaging, and file sharing may be provided. Media, including video, text, and images, may be selected and sent to participants while they are on hold or during an active call session.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/474,467, filed Sep. 14, 2021, entitled SYSTEM AND METHOD FORESTABLISHING AND MANAGING MULTIPLE INDEPENDENT CALL SESSIONS FROM ACENTRALIZED CONTROL INTERFACE (Atty. Dkt No. DAMA60-35333), which claimsthe benefit of U.S. Provisional Application Ser. No. 63/077,892, filedon Sep. 14, 2020, and entitled SYSTEM AND METHOD FOR ESTABLISHING ANDMANAGING MULTIPLE INDEPENDENT CALL SESSIONS FROM A CENTRALIZED CONTROLINTERFACE. U.S. application Ser. No. 17/474,467 also claims the benefitof U.S. Provisional Application Ser. No. 63/176,419, filed on Apr. 19,2021, entitled SYSTEM AND METHOD FOR HIGHLY SCALABLE BROWSER-BASEDAUDIO/VIDEO CONFERENCING, which are incorporated by reference herein inits entirety.

This application claims the benefit of U.S. Provisional Application Ser.No. 63/176,419, filed on Apr. 19, 2021, and entitled SYSTEM AND METHODFOR HIGHLY SCALABLE BROWSER-BASED AUDIO/VIDEO CONFERENCING, which isincorporated by reference herein in its entirety.

BACKGROUND

The manner in which communication sessions with remote parties occur iscurrently limited in functionality and flexibility. Accordingly, what isneeded are a system and method that addresses these issues.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding, reference is now made to thefollowing description taken in conjunction with the accompanyingDrawings in which:

FIGS. 1A and 1B illustrate embodiments of environments within which acontrol device may manage separate communication sessions with one ormore client devices;

FIG. 1C illustrates one embodiment of a virtual waiting room and callsystem within the environment of FIG. 1A;

FIGS. 1D-1H illustrate embodiments of different arrangements of thecomponents of the virtual waiting room and call system of FIG. 1C;

FIG. 2-5 illustrate embodiments of a graphical user interface (GUI) thatmay be used by the control device of FIGS. 1A and 1B;

FIG. 6A illustrates one embodiment of a GUI by which call requests maybe assigned to a virtual waiting room;

FIG. 6B is a flow chart illustrating an embodiment of a process that maybe executed by a control device;

FIG. 6C is a flow chart illustrating an embodiment of a process that maybe executed by a client device;

FIGS. 7A-7E illustrate processes by which media may be injected into asession, either directly or via a URL, to be played when the session isactive or on hold;

FIGS. 8-10 are flow charts illustrating embodiments of processes thatmay be executed by a control device to provide media for display on aclient device;

FIG. 11 is a flow chart illustrating an embodiment of a process that maybe executed by a client device;

FIGS. 12A and 12B illustrate embodiments of a medical facilityenvironment within which a control device may manage communicationsessions with multiple client devices;

FIGS. 12C and 12D illustrate embodiments of FIGS. 1A-1C, 12A, and 12Bthat use one or more servers for conference calls;

FIGS. 13A-17F are embodiments of sequence diagrams illustrating variousprocesses that may be executed within the environments of FIGS. 1A, 1B,and 12A-12F;

FIGS. 18-26 illustrate embodiments of a GUI that may be used by acontrol device;

FIGS. 27-29 illustrate embodiments of a GUI that may be used by a clientdevice; and

FIG. 30 is a simplified diagram of one embodiment of a computer systemthat may be used in embodiments of the present disclosure as a controldevice, a client device, and/or a server.

DETAILED DESCRIPTION

It is understood that the following disclosure provides many differentembodiments or examples. Specific examples of components andarrangements are described below to simplify the present disclosure.These are, of course, merely examples and are not intended to belimiting. In addition, the present disclosure may repeat referencenumerals and/or letters in the various examples. This repetition is forthe purpose of simplicity and clarity and does not in itself dictate arelationship between the various embodiments and/or configurationsdiscussed.

Referring to FIGS. 1A and 1B, embodiments of environments areillustrated within which various aspects of the present disclosure maypracticed. Referring specifically to FIG. 1A, an environment 100includes a control device 102 that has established communicationsessions with a client device 104, a client device 106, and a clientdevice 108. The control device 102 is coupled to the client device 104via a session 110, to the client device 106 via a session 112, and tothe client device 108 via a session 114. Although only three clientdevices are illustrated, it is understood that any number of clientdevices may be in communication with the control device 102, subject totechnical limitations such as bandwidth, processing power, and similarfactors.

FIG. 1B illustrates an environment 116 in which the control device 102has established only the communication session 110 with the clientdevice 104. It is understood that the environments 100 and 116 may beviewed as the same environment in different states. For example, thecontrol device 102 of FIG. 1A may drop two sessions to reach the singlesession of FIG. 1B. Similarly, the control device 102 of FIG. 1B may addtwo sessions to reach the three sessions of FIG. 1A.

The control device 102 and client devices 104, 106, and 108 may bemobile devices (e.g., tablets, smartphones, personal digital assistants(PDAs), or netbooks), laptops, desktops, workstations, smarttelevisions, and/or any other computing device capable of receiving andsending electronic communications via a wired or wireless networkconnection. Such communications may be direct (e.g., via a peer-to-peernetwork, an ad hoc network, or using a direct connection), indirect,such as through a server or other proxy (e.g., in a client-servermodel), or may use a combination of direct and indirect communications.

While multiple sessions may be combined into a single session (e.g., asa conference call or a whiteboard sharing session), it may be desirablein some scenarios described herein to maintain the sessions 110, 112,and 114 as independent sessions. Accordingly, with independent sessions,only the control device 102 has access to the session established witheach of the client devices 104, 106, and 108, and the client devices donot have access to the sessions of the other client devices. In otherwords, neither the client device 106 nor the client device 108 mayaccess the session 110 unless the control device 102 specifically mergesthe sessions, and such functionality may or may not be available to thecontrol device depending on the particular configuration of the controldevice and/or client devices. For example, such functionality may bedisabled to prevent a user of the control device 102 from inadvertentlymerging sessions. This may be of particular value in environments whereprivacy requirements are high and possibly regulated by law, such astelemedicine and counseling environments.

In other embodiments, a client device may be provided the neededfunctionality to invite another device to a session. In suchembodiments, the other device may not be part of an existing session,but would be invited to participate in the session. Each session may beend-to-end encrypted for both signaling and media. This enables sessionsto be channeled through third-party servers without compromising privateinformation.

A session may be accessed by the control device 102 and client devicesin various ways. For example, a session may be accessed in a web basedmanner via a web browser (e.g., based solely on a browser's nativecapabilities or using scripting, an applet, and/or other browser relatedfunctionality), or may be accessed in an application based manner usinga downloaded application that is installed and/or used to execute atemporary file. It is understood that the control device and clientdevice may access a session in different ways, such as an installedapplication for the control device and browser access for the clientdevice or vice versa.

The terms “control device” and “client device” as used herein do not inthemselves dictate a server/client or other particular relationshipbetween the control and client devices. As will be seen in followingexamples, the terms are generally used in a relational manner toindicate that the control device is controlling one or more sessions viaa control interface, and each client device is communicating with thecontrol device via one of those sessions. In some embodiments, thecontrol device 102 may have access to functionality not available to theclient device, such as the ability to inject videos and/or other mediainto the session when the session is ongoing or placed on hold. Inaddition, it is understood that each of the control device and clientdevices are generally used for interactions with a user, and the userinformation may be used or displayed in any given example herein. Forexample, the control device 102 may be identified as such, or may beidentified by a user name or other designation.

For purposes of example, each of the communication sessions 110, 112,and 114 is described with respect to a hybrid peer-to-peer model, butthe present disclosure is not limited to such models. It is understoodthat each of the sessions 110, 112, and 114 may include multipleconnections or channels for different functions, such as aconnection/channel for an audio/visual call and anotherconnection/channel for document sharing. Alternatively, a singleconnection/channel may be used to provide multiple functions. Examplesof hybrid peer-to-peer communications and functionality that may be usedherein are disclosed in U.S. Pat. Nos. 7,656,870; 7,623,516; 7,623,476;7,933,260; 8,050,272; 8,218,444; 8,446,900; 8,694,587; 8,892,646;9,027,032; and 9,356,997.

From a technical perspective, each device may act as a client, a server,and/or as something else depending on the underlying technologicalframework used to establish and maintain the communication sessions. Forexample, in a hybrid peer-to-peer framework, both the control device 102and a client device may take the roles of a server and a client,depending on which way information is being transferred at a given time.In a traditional client-server framework, both the control device 102and the client device may be clients interacting with a server that isseparate from either device. Accordingly, it is understood that thefunctionality described herein may be translated to varioustechnological frameworks, with appropriate method steps and messageflows that correspond to the particular framework in which the presentdisclosure is implemented.

Referring to FIG. 1C, one embodiment of a communications framework 120is illustrated with a virtual waiting room (VWR) management interface122. The VWR 122 may be used to manage session resources 124 a-124 cthat handle the actual communications with the client devices 104, 106,and 108, respectively. For example, the session resources 124 a-124 cmay be used for the allocation of memory, communication sockets, dataprioritization, and/or other resources that are used to send and receivedata. The VWR management interface 122 may provide a visual view of theend result (e.g., the availability and/or existence of multiplesessions) and provide management components such as interactive elements(e.g., buttons) for placing calls on hold, providing users with uniformcommunication and collaboration (UCC) functionality, and similarfeatures. In some embodiments, the VWR management interface 122 may befurther divided to provide a backend interface that may be used to sendinvitations to the VWR, schedule calls, and perform similar functions,and a frontend interface that may then access the calls waiting in theVWR. The division and organization of such functionality may differbased on the particular implementation.

Referring to FIGS. 1D-1G, embodiments of different arrangements of thecommunications framework 120 of FIG. 1C are illustrated. In FIG. 1D, thecommunications framework 120 is located on the control device 102. InFIG. 1E, the communications framework 120 is located on one or moreservers 126 and accessed by the control device 102. In FIG. 1F, the VRMmanagement interface 122 is located on the control device 102, and thesession resources 124 a-124 c are located on the server 126. In FIG. 1G,the VRM management interface 122 is divided into a frontend portion 122a and a backend portion 122 b. The frontend portion 122 a is located onthe control device 102, and the backend portion 122 b and sessionresources 124 a-124 c are located on the server 126. In FIG. 1H, the VRMmanagement interface 122 is divided into a frontend portion 122 a and abackend portion 122 b, both of which are located on the server 126 alongwith the session resources 124 a-124 c. In some embodiments, the controldevice 102 may also be directly coupled to the backend portion 122 b, asshown in FIG. 1H. The server 126 may represent one or more virtualand/or physical servers in FIGS. 1E-1H. It is understood that manydifferent arrangements of functionality are possible, and FIGS. 1C-1Hare meant for purposes of example only and are not intended to belimiting.

The communications framework 120 may enable all of the client devices104, 106, and 108 to request access to the control device 102 via asingle, identical uniform resource locator (URL) or other uniformresource identifier (URI). Because the communications framework 120enables sessions to be established and managed together, it can identifydifferent session requests and set each session up independently eventhough all requests are received via the same URL. This means thateither static or dynamic URL allocation may be used, and a single URLcan be send to multiple invitees for independent sessions. It isunderstood that this is not required and that unique URLs may be used ifdesired.

Referring to FIG. 2 , one embodiment of a graphical user interface (GUI)200 is illustrated for the control device 102 of FIGS. 1A and 1B. TheGUI 200 may represent, for example, the VRM management interface 122 ofFIGS. 1C and 1F, and the frontend VRM management interface 122 a ofFIGS. 1G and 1H.

In the present example, the GUI 200 may be used to control multiplecommunication sessions, such as the sessions 110, 112, and 114.Individual sessions may be added or dropped without affecting the statusof the other sessions. Depending on the embodiment, a limited number ofsession slots may be available, or sessions may be added untilconstrained by technological or other factors (e.g., memory, bandwidth,or available time slots). The GUI 200 may be web-based and accessible bycontrol device 102 via a web browser, or may be application based andaccessible by the control device using a downloaded application that iseither installed or used to execute a temporary file.

In the present example, the sessions are with individuals using theclient devices 104, 106, and 108. The sessions are separate and remainso unless merged by the control device 102. The client devices 104, 106,and 108 may not have the ability to unilaterally merge with anothersession, but may have the ability to approve or reject a merge initiatedby the control device 102. In addition, the control device 102 and/orthe client devices 104, 106, and 108 may have the ability to create anew session (e.g., rather than merging an existing session) and add thatsession to an existing session.

The GUI 200 provides a session list for the user of the device 102, witha session indicator 202 a for the session 110, a session indicator 202 bfor the session 112, and a session indicator 202 c for the session 114.In the present example, each session indicator 202 a-202 c includes aname for the individual communicating via the respective client device.

The GUI 200 may further include controls for each of the sessions 110,112, and 114, with a control 204 a for session 110, a control 204 b forsession 112, and a control 204 c for session 114. The controls 204 a-204c enable a currently active session to be put on hold (e.g., the control204 b) or may be used to switch to a currently held session (e.g., thecontrols 204 a and 204 c). Switching to a session that is on hold mayautomatically put the current session hold. Putting the current sessionon hold may retain the current session as the active session (e.g., mayplace the current session on hold without switching to another session).Additional controls 206 a and 206 b may be used to merge existingsessions.

A video display window 208 may be provided for a video call in theactive session. As the current session is with Rashmi, her name may beprovided in the window. Controls such as a speaker button 210, a mutebutton 212, and a video button 214 may be provided for control over theaudio and video aspects of the call.

A screenshare button 216 may be provided to enable screen sharing. Asend button 218 may be provided to send data in the current session. Forexample, selection of the send button 218 may open a dialog box or menufrom which one or more files, URLs, and/or other data may be selectedfor transmission to the client device separate from, or embedded into,the ongoing call. The terms “file” and “media” as used herein includeany type of text, image, model (e.g., a 3-D model or information neededto render/manipulate such a model), audio, and/or video file orcombination of files, and may include files having multiple media types(e.g., an audio/video file or a text document with inline or linkedimages).

The data may be local on the control device 102, may be retrieved from astorage location (e.g., a server or database) prior to being sent by thecontrol device, or may be accessed by the client device from a storagelocation (e.g., a server or database) using a link or other accessmechanism provided by the control device. The data may be send as adiscrete file or streamed, depending on the particular file and transfermechanism. Data may also be provided via a link to the client device.

The send process may be accomplished in different ways. The system maybe limited using various parameters or may dynamically adjust how datais sent based on factors such as available bandwidth and processingpower, the type of data to be sent, whether the call is on hold oractive, and similar factors.

Referring to FIG. 3 , for example, a file may be sent during an activecall so that the control and/or client devices may display the filewhile the call is ongoing. The file may be an image, a text file, avideo file, or any other file type or combination of types. The GUI 200may provide a file view window 300 with a file viewing space 302.Depending on the size of the file, a slider 304 and slider control 306may be used to scroll through the file vertically. A horizontal control(not shown) similar to the slider 304 for horizontal movement may beprovided in some embodiments. A 3D control (not shown) may be providedto manipulate 3D files. One or more tabs or other selection components,such as tabs 308 a-308 d, may be provided to access multiple files.

In other embodiments, data may be sent while the call is on hold,enabling the client device to display the video file or other file whilewaiting for the control device 102 to restore the session to activestatus. One embodiment includes sending/beginning to stream a file andthen placing the session on hold (the file may still besending/streaming while on hold in some examples). Another embodimentincludes placing the session on hold and then sending/streaming the fileto the client device. Still another embodiment includes sending a URL tothe client device before or after placing the session on hold, and theclient device selects the URL to download or stream the file. Such datatransmissions may require user acceptance/authorization or mayautomatically play on the client device.

In some embodiments, a URL or a file may be sent or streamed to theclient device from a preset list of available options before or afterthe session is put on hold. The selection may be manual or automatic,with automatic selections having a variety of options such aspredefined, prioritized, based on the current session's context, and/orrandom. In some embodiments, user engagement may be tracked to recordwhether the file was viewed and, if so, how much of the file was viewed.In other embodiments, the file may simply be transferred withouttracking user engagement.

Each feature of the GUI 200, including the session indicators 202 a-202c and controls 204 a-204 c, 206 a, and 206 b may include text, colors,border modifications (e.g., dotted lines, thicker or multiple lines,etc.), and/or other variations to indicate the status of the associatedsession and/or available functionality. For example, the active sessionof session indicator 202 b may be outlined or filled with green, and theinactive sessions of session indicators 202 a and 202 b may be outlinedor filled with yellow or red. Additionally or alternatively, some or allsection indicators may be shaded in color, may display symbols, etc., toindicate the status of their respective sessions. Functionality may begrayed out or removed if not available. Furthermore, the amount of timeon hold may be indicated via a visible timer, changes in the sessionindicator (e.g., slowly turning from yellow to red), and/or using othermethods.

In addition, it is understood that many different arrangements ofvarious elements illustrated on the GUI 200 may be used. Elements may beremoved, added, combined, further divided, and positioned differently.Accordingly, it is understood that many different graphical presentationand implementation techniques may be used with the GUI 200 and thepresent disclosure is not intended to be limited to those explicitlyillustrated or described.

Referring to FIG. 4 , one embodiment of the GUI 200 of FIG. 3 is shownwith session 110 merged with session 112, so a video window 208 b forSiva is now present. Document sharing and other functionality maycontinue with both Siva and Rashmi.

Referring to FIG. 5 , one embodiment of the GUI 200 of FIG. 2 is shownwith a single call session. Some functionality displayed with respect toFIG. 2 is not displayed in this example since only the single call ispresent.

Referring to FIG. 6A, one embodiment of a GUI 600 is illustrated. TheGUI 600 may represent, for example, the backend VRM management interface122 b of FIGS. 1G and 1H. In the present example, the GUI 600 includesone or more possible call queues 602. The call queues include a queuefor a control device 604 a and a control device 604 b (or namesassociated with those devices). The queue for the control device 604 hasfour available slots, with slots 606 a-606 c indicated as being busy,and slot 606 d indicated as being available. The control device 604 bhas no available slots and is indicated as being offline.

The GUI 600 is displaying incoming requests 608 a-608 c. The incomingrequest 608 a has been selected, showing additional selectionpossibilities such as an additional information button 610, anassignment button 612, and a schedule button 614. The incoming requestbutton 610 may provide additional information about the caller, whethera particular queue should be used (e.g., an appointment for a particulardoctor rather than a general call for first available), and similar userspecific details. The assignment button 612 may be used to assign thecall to a particular available slot (e.g., the slot 606 d). This willplace the request in the virtual waiting room for the control device 604a. The schedule button 614 may be used to schedule a later call, such aswith the control device 604 b when it comes online. In some embodiments(not shown), the control device 604 b may have slots available forfuture calls, and the incoming request 608 a may be assigned to one suchslot.

Referring to FIG. 6B, a flow chart illustrates one embodiment of aprocess 620, executable by the control device 102 or a component of thecommunications framework 120, by which client devices may be invited toa virtual waiting room and provided media to display while waiting. Instep 622, the control device 102 or a component of the communicationsframework 120 sends an invitation to a client device. In step 624, theincoming client device is placed in the virtual waiting room. In step626, media (e.g., audio, video, text, images, and/or other types offiles) or URLs representing media locations may be sent to the waitingclient device. In some embodiments, step 626 may occur as part of theinvitation of step 622, between steps 622 and 624, or with step 624.

In step 628, a list of client devices in the virtual waiting room may bedisplayed. In step 630, a selection of a particular client device isreceived with any additional options (e.g., whether the session is to beinitiated as a chat session, audio only call, or video call). In step632, the session is initiated. Step 632 will generally end or suspendthe media on the client device. Playback may continue if the session isagain placed on hold, if the user of the client device indicates theywish to continue, and/or based on other parameters.

Referring to FIG. 6C, a flow chart illustrates one embodiment of aprocess 640, executable by a client device, by which the client devicemay be invited to a virtual waiting room and provided media to displaywhile waiting. In step 642, the client device 102 receives an invitationto the virtual waiting room and joins in step 644. In step 646, media(e.g., audio, video, text, images, and/or other types of files) or URLsrepresenting media locations are received by the waiting client device.In some embodiments, step 646 may occur as part of the invitation ofstep 642, between steps 642 and 644, or with step 644. In step 648, theclient device plays the media while waiting for the session to begin.

Referring to FIGS. 7A-7E, embodiments are illustrated where the controldevice 102 is sending media (e.g., audio, video, text, images, and/orother types of files) to one or more client devices. With respect toeach of FIGS. 7A-7E, as described previously, the sending of mediaand/or URLs may occur with or without putting the correspondingcommunication session(s) on hold. This process enables educationalmaterial, advertisements, and/or other material to be displayed on aclient device while the session is active or on hold.

The particular information may be selected based on the context of thesession. For example, if the session is a medical consultation, themedia may be related to available diagnostic procedures, potentiallyapplicable drugs, and/or other information that may be used to informand/or educate the patient while waiting for the session to continue. Ifthe session is for discussing the purchase or rental of a product orservice, the media may be related to options, upgrades, alternatives,availability, special deals, and/or other information that may beapplicable to the consumer. If the session is a consultation session foradvice (e.g., beauty advice, counseling, recruiting, or professionaldevelopment), the media may be related to various options, products,subscriptions, and/or other information that may be applicable to theindividual seeking advice. The media may play automatically or mayrequire the user of the client device to accept the media.

In FIG. 7A, the media is sent directly to the client device 104. In FIG.7B, the media is sent directly to the client devices 104 and 106. Theclient devices 104 and 106 may be in independent sessions with thecontrol device 102 or may be in a conference call together. The mediasent to each of the client devices 104 and 106 may be identical ordifferent.

In FIG. 7C, a URL is sent to the client device 104 and the client device104 then retrieves the media from a server 702 using the URL. In FIG.7D, a URL is sent to each of the client devices 104 and 106, and eachclient device 104 and 106 then retrieves the media from the server 702using the URL. The client devices 104 and 106 may be in independentsessions with the control device 102 or may be in a conference calltogether. The URL sent to each of the client devices 104 and 106 may beidentical or different, and may direct the client devices to the sameserver or to different servers.

In FIG. 7E, the media is sent directly to the client device 106, and aURL is sent to the client device 104, which then retrieves the mediafrom the server 702 using the URL. The client devices 104 and 106 may bein independent sessions with the control device 102 or may be in aconference call together. The media directly sent and represented by theURL may be identical or different.

Referring to FIG. 8 , a flow chart illustrates one embodiment of aprocess 800 by which media may be sent during a period in which acommunication session is on hold or to be placed on hold. In step 802,the control device 102 or a component of the communications framework120 identifies that the current session has been placed on hold. In step804, media options are provided to a user of the control device 102. Instep 806, a media selection is received. In step 808, the media or URLis injected into the media session for delivery to the client device104. In some embodiments, step 802 may occur after step 804, 806, or808.

Referring to FIG. 9 , a flow chart illustrates one embodiment of aprocess 900 by which media may be sent during a period in which acommunication session is on hold or to be placed on hold. In step 902,the control device 102 or a component of the communications framework120 identifies that the current session has been placed on hold. In step904, media is automatically selected using a predefined priority list, arandomized selection process, or another method. In step 908, the mediaor URL is injected into the media session for delivery to the clientdevice 104. In some embodiments, step 902 may occur after step 904 or906.

Referring to FIG. 10 , a flow chart illustrates one embodiment of aprocess 1000 by which previously sent media may have been fully consumedor a request may be received by the client device 104. In step 1002, thecontrol device 102 or a component of the communications framework 120detects that previously sent media has ended and/or receives a requestfor media from the client device 104. In step 1004, media is obtained asdescribed with respect to steps 804 and 806 of FIG. 8 or step 904 ofFIG. 9 . In step 1006, the media or URL is injected into the mediasession for delivery to the client device 104. In some embodiments, step1002 may occur after step 1004.

Referring to FIG. 11 , a flow chart illustrates one embodiment of aprocess 1100 by which media may be received for use by the client device104 during a period in which a communication session is on hold or to beplaced on hold. In the present example, the session 110 has been placedon hold prior to step 1102, but the hold may be placed at any timebefore the execution of step 1110 in other embodiments.

In step 1102, media or a URL is received. Although not shown, anauthorization step may be present in some embodiments that enables auser of the client device 104 to reject the media/URL. If this occurs,the method 1100 may move directly to step 1116. The followingdescription of FIG. 11 assumes that authorization has occurred or wasnot needed.

If media is received in step 1102 as determined in step 1104, the methodproceeds to step 1108 where the media is played. If a URL is received,the method 1100 moves to step 1106 to obtain the media before moving tostep 1108. If the hold remains in place as determined in step 1110, themedia may continue playing. It is understood that step 1110 may be aninterrupt rather than a determination, in which case the media may playuntil an interrupt occurs.

If the hold is removed, the media is ended/suspended in step 1112 andthe method 1100 moves to step 1118 where the call is continued. Theremay be a closing notice, such as a an audio or text “Thank you” as themedia is being closed. If the hold remains, the method 1110 maydetermine if the media has ended (e.g., finished playing if a video oraudio file). If the media has not ended, the method 1100 may return tostep 1108 and continue playing the media. If the media has ended, themethod 1100 waits for the removal of the hold in step 1116. Once thehold is removed, the call continues in step 1118. In some embodiments,the client device 104 may request additional media in following step1114 or may receive additional media as indicated in FIG. 10 .

In some embodiments described in the present disclosure, analytical datamay be compiled. For example, if media is played while sessions are onhold, the media name, number of times the media is played, whether themedia was paused or reversed for additional viewing, the control devicethat selected the media, and similar information may be compiled. Suchdata may be used to determine effectiveness of particular media, gaugeinterest, identify effectiveness or use patterns relative to othermedia, etc.

Referring to FIG. 12A, in one embodiment, the components of theenvironment 100 of FIG. 1A are illustrated in a medical environment1200. Accordingly, the environment 1200 includes a medical facility1202, which may be all or a portion of a hospital, clinic, doctor'soffice, and/or any other medical facility. It is understood that themedical facility 1202 may operate within a larger facility, such as aclinic located within a hospital or on a hospital's grounds. The medicalfacility 1202 may include or otherwise access a server 1204, a localelectronic storage 1206 (e.g., a database), and/or a remote electronicstorage 1208. In some embodiments, the server 1204 may incorporate oneor both of the local/remote electronic storages 1206/1208 into theserver. In other embodiments, the server 1204 may be located outside ofthe medical facility 1202, such as a server provided by a hospital for aclinic's use or a cloud based server.

The server 1204 may host at least some of the technology needed for thecommunications and virtual waiting room functionality to operate. Forexample, if the technological framework is a hybrid peer-to-peer system,the server 1204 may function as the server that performs operationsneeded for such a system as described in previously referenced U.S. Pat.Nos. 7,656,870; 7,623,516; 7,623,476; 7,933,260; 8,050,272; 8,218,444;8,446,900; 8,694,587; 8,892,646; 9,027,032; and 9,356,997.

With additional reference to FIG. 12B, another embodiment of the medicalenvironment 1200 is illustrated as an environment 1220. In this example,the server 1204 and electronic storage 1208 (which may be combined withor separate from the server 1204) are remote from the medical facility1202. For example, the server 1204 and electronic storage 1208 may beprovided via cloud services that are owned or leased by the medicalfacility 1202.

With respect to both FIGS. 12A and 12B, if the server 1204 andlocal/remote electronic storages 1206/1208 are under control of themedical facility 1202, oversight of legal and ethical considerations(e.g., compliance with the Health Insurance Portability andAccountability Act (HIPAA) and other applicable laws and ethical rules)may be simplified as third parties will not be in control or have accessto any regulated information. It is noted that the server 1204 andelectronic storage 1208 of FIG. 12B may be viewed as being under controlof the medical facility as long as the medical facility personnel arethe only ones to have access to the remote components and/or regulatedinformation that is processed or stored by the server/electronicstorage. In addition, as each session may be end-to-end encrypted forboth signaling and media, sessions can be channeled through third-partyservers without compromising private information.

If media is to be injected into a session while the session is active oron hold, a URL or list of URLs may be provided to the server 1204, oneor both of the electronic storages 1206/1208, and/or to the controldevice 102. This avoids the need to store the media within the medicalfacility, although such storage may be used for at least some of themedia in some embodiments. Media may also be stored on a remote server1210 that may or may not be associated with the medical facility 1202.

In some embodiments, the control device 102 may not be located withinthe medical facility some or all of the time. For example, if thecontrol device 102 is a tablet, a doctor within the medical facility mayuse the tablet from home. In other embodiments, the control device 102may be limited in functionality based on a geofence, time parameters,and/or other restrictions.

The previously described functionality and interfaces may be directed tothe medical environment 1200. For example, the control device 102 may beassociated with a doctor, a nurse practitioner, or other medicalpersonnel (all personnel may be generally referred to herein as “doctor”for purposes of illustration). The client devices 104, 106, and 108 maybe associated with patients of the doctor. The sessions 110, 112, and114 are to remain separate as each may involve the transfer ofconfidential and sensitive health data in the form of verbal and visualcommunications, text chat, documents, images (e.g., CAT scans andX-rays), etc.

As described previously, additional client devices may be added to asession in a conference call if desired. For example, if the patient isa minor, a parent or guardian may be present. Family members, friends,other medical personnel (e.g., specialists), and/or other individualsmay be added to a session. If needed, consent forms may be sent to usersof client devices to ensure that proper documentation procedures arefollowed, just as they would be for in-office visits. Accordingly, bysending and receiving online forms, scans or images of signatures, andsimilar electronic documents, the present disclosure enables establishedprocedures to be maintained even in virtual consultations.

Reminders may be used if documents are needed, or functionality may beblocked until certain steps are taken. For example, audio and video maynot be available to a client device unless a consent form is received.It is understood that such constraints are customizable and may not berequired unless desired. Patient education (PE) information,advertisements, and other information may be sent to the client devicesas described in previous embodiments, either directly (e.g., as shown inFIG. 7A) or indirectly (e.g., as shown in FIG. 7B).

Referring to FIG. 12C, one embodiment of an environment 1230 illustratescomponents of the environments 1200 (FIG. 12A) and 1220 (FIG. 12B),although it is understood that the components of the environment 1230may be used in any many different implementations of the presentdisclosure. In the present example, the server 1204 is represented by agateway server 1204 a and a conference server 1204 b, which may includea multipoint control unit (MCU) designed to facilitate conference calls.In other embodiments, the functionality of the servers 1204 a and 1204 bmay be performed by a single server, in which case only the singleserver would be present. In the present example, the control device 102is in one-to-one sessions (e.g., an active call, on hold, or waiting)with each of the client devices 104, 106, and 108. Accordingly, theconference server 1204 b is not needed. The gateway server 1204 ahandles signaling for the control device and client devices, while mediamoves directly between the control device 102 and each client device.

Referring to FIG. 12D, one embodiment of an environment 1240 illustratesthe components of the environment 1230 in a conference call scenario.The control device 102 is involved in a conference call with the clientdevices 104, 106, and 108. The gateway server 1204 a handles signalingfor the control device and client devices, while media moves between thecontrol device and client devices through the conference server 1204 b.

FIGS. 13A-17F are embodiments of sequence diagrams illustrating variousprocesses that may be executed within the environments of FIG. 1A, 1B,and 12A-12D. Although the particular examples of FIGS. 13A-17F maycontain details relevant specifically to the medical environment 1200 ofFIGS. 12A and 12B (e.g., references to doctors, patients, patienteducation material, and other medical oriented terminology), the varioussteps may be applied to many different environments, including sessionsbetween consultants and clients, teachers and students, and salespeopleand customers.

It is understood that the sequence diagrams described herein illustratevarious exemplary functions and operations that may occur within variouscommunication environments. It is understood that these diagrams are notexhaustive and that various steps may be excluded from the diagrams toclarify the aspect being described. For example, it is understood thatsome actions, such as network authentication processes andnotifications, may have been performed prior to the first step of asequence diagram. Such actions may depend on the particular type andconfiguration of a particular device, including how network access isobtained (e.g., cellular or WLAN access). Other actions may occurbetween illustrated steps or simultaneously with illustrated steps,including network messaging for call maintenance (including handoffs),communications with other devices (e.g., email, text messages, and/orvoice calls (including conference calls)), and similar actions. Inaddition, is it understood that single messages may be illustrated ordescribed, and such messages may actually represent a series ofmessages.

Referring to FIGS. 13A-13D, one embodiment of a sequence diagramillustrates a process 1300 by which the control device 102 (representinga doctor) within the medical environment of FIGS. 12A and 12B maycommunicate with client devices 104 (representing a first patient) and106 (representing a second patient) in separate sessions. It isunderstood that some steps (such as file sharing or screen sharing) mayoccur at different times than those illustrated in FIGS. 13A-13D (or notat all), and that such steps are included to show how general messageflows for various features may be incorporated.

For purposes of simplicity in this and following embodiments, thecontrol device 102 may be hosting or otherwise providing the waitingroom. In embodiments where a separate device or server is providing thewaiting room (e.g., the gateway server 1204 a of FIGS. 12C and 12D),additional messages would be passed between the control device 102 andthe hosting device/server with information about waiting patients, etc.,and client devices would interact with the hosting device/server asneeded.

Referring specifically to FIG. 13A, in step 1302, the control device 102logs into a server 1204, which is the server 1204 of FIG. 12A or 12B inthe present example. In steps 1304 and 1306, respectively, the controldevice 102 sends an invitation to the client devices 104 and 106 viaemail, SMS, or other communication channels. In steps 1308 and 1310,respectively, the client devices 104 and 106 join the doctor's waitingroom (which may be specific to the doctor or may be a general waitingroom in the medical facility). In step 1312, the server 1204 notifiesthe control device 102 of the available status of the client devices 104and 106. In steps 1314 and 1316, respectively, the server 1204 notifiesthe client devices 104 and 106 of the available status of the controldevice 102. In steps 1318 and 1320, respectively, the client devices 104and 106 notify the control device 102 that they are in the waiting room.

Referring specifically to FIG. 13B, in step 1322, the control device 102sends patient education material to the client device 106, which theclient device 106 displays to its user in step 1324. As this process hasbeen described in previous embodiments, it is not further detailed here.In step 1326, the control device 102 starts a session (e.g., an audio orvideo call) with the client device 104. In step 1328, the control device102 notifies the server 1204 that it is on a call, and the server 1204updates the control device's status to the client device 106 in step1330. At this time, there is an active call between the control device102 and the client device 104 as indicated by step 1332. In step 1334,the control device 102 places the active call with the client device 104on hold. In step 1336, the control device 102 sends patient educationmaterial to the client device 104, which the client device 104 displaysto its user in step 1338.

Referring specifically to FIG. 13C, in step 1340, the control device 102removes the hold on the call with the client device 104. When the holdis removed, the display of the media sent to the client device 104 instep 1336 may be paused or otherwise suspended. At this time, the activecall between the control device 102 and the client device 104 continuesas indicated by step 1342. In step 1344, the control device 102 sharesone or more files with the client 104 during the active call. In step1346, the active call between the control device 102 and the clientdevice 104 is ended. In step 1348, the control device 102 updates itsstatus with the server 1204 as available, and the server 1204 notifiesthe client device 106 of the updated status in step 1350. In step 1352,the control device 102 instructs the server 1204 to disconnect theclient device 104 from the waiting room, and the server 1204 notifiesthe client device 104 of the disconnection in step 1354.

Referring specifically to FIG. 13D, in step 1356, the control device 102sends a chat message to the client device 106 while the client device isdisplaying the media. For example, the chat message may inform the userof the client device that the doctor is now ready or will be ready in aparticular amount of time (e.g., five minutes). In step 1358, thecontrol device 102 starts the call with the client device 106. At thistime, there is an active call between the control device 102 and theclient device 106 as indicated by step 1360. In step 1362, the controldevice 102 sends patient education material to the client device 106during the active call, which the client device 106 displays to its userin step 1364. In step 1366, the control device 102 shares its screenwith the client 106 during the active call. In step 1368, the controldevice 102 shares one or more files with the client 106 during theactive call.

In step 1370, the active call between the control device 102 and theclient device 106 is ended. In step 1372, the control device 102 updatesits status with the server 1204 as available. In step 1374, the controldevice 102 instructs the server 1204 to disconnect the client device 106from the waiting room, and the server 1204 notifies the client device106 of the disconnection in step 1376.

Referring to FIGS. 14A and 14B, one embodiment of a sequence diagramillustrates a process 1400 by which the control device 102 (representinga doctor) within the medical environment of FIGS. 12A and 12B maycommunicate with client devices 104 (representing a first patient) and106 (representing a second patient) in separate sessions. It isunderstood that some steps (such as file sharing or screen sharing) mayoccur at different times than those illustrated in FIGS. 14A and 14B (ornot at all), and that such steps are included to show how generalmessage flows for various features may be incorporated.

Referring specifically to FIG. 14A, in step 1402, the control device 102logs into a server 1204 a, which is the server 1204 of FIG. 12A or 12Bin the present example. In step 1404, the control device 102 sends aninvitation to the client device 104 via email, SMS, or anothercommunication channel. In step 1406, the client device 104 joins thedoctor's waiting room (which may be specific to the doctor or may be ageneral waiting room in the medical facility). In step 1408, the server1204 a notifies the control device 102 of the available status of theclient device 104. In step 1410, the server 1204 a notifies the clientdevice 104 of the available status of the control device 102. In step1412, the client device 104 notifies the control device 102 that it isin the waiting room. In step 1414, the control device 102 begins a callsession with the client device 104. At this time, there is an activecall between the control device 102 and the client device 104 asindicated by step 1416.

In step 1418, the control device 102 notifies the server 1204 a to beginrecording. In step 1420, the server 1204 a instructs the server 1204 b(which may be the same server as the server 1204 a or a differentserver) to set up a recording session for the control device 102 and theclient device 104. In steps 1422 and 1426, respectively, the server 1204a sends a start recording message to the control device 102 and clientdevice 104 to begin recording. This message may contain informationneeded to send data to the server 1204 b.

Referring specifically to FIG. 14B, in steps 1426 and 1428,respectively, the control device 102 and the client device 104 sendsession media to the server 1204 b to be recorded. In step 1530, thecontrol device 102 shares its screen with the client device 104, whichmay happen many different times during the call. In step 1432, thecontrol device 102 notifies the server 1204 a to stop recording. In step1434, the server 1204 a instructs the server 1204 b to stop therecording session. In steps 1436 and 1438, respectively, the server 1204a sends a stop recording message to the control device 102 and clientdevice 104.

In 1440, the active call between the control device 102 and the clientdevice 104 is ended. Although not shown, the control device 102 mayupdate its status with the server 1204 a as available. In step 1442, thecontrol device 102 instructs the server 1204 a to disconnect the clientdevice 104 from the waiting room, and the server 1204 a notifies theclient device 104 of the disconnection in step 1444. In step 1446, thecontrol device 102 may retrieve the recording from the server 1204 b andplay back the session in step 1448.

Referring to FIGS. 15A-15F, one embodiment of a sequence diagramillustrates a process 1500 by which the control device 102 (representinga doctor) within the medical environment of FIGS. 12A and 12B maycommunicate with client devices 104 (representing a first patient) and106 (representing a second patient) in a conference call. It isunderstood that some steps (such as file sharing or screen sharing) mayoccur at different times than those illustrated in FIGS. 15A-15F (or notat all), and that such steps are included to show how general messageflows for various features may be incorporated.

Referring specifically to FIG. 15A, in step 1502, the control device 102logs into a server 1204 a, which is the server 1204 a of FIGS. 12C and12D in the present example. In steps 1504 and 1506, respectively, thecontrol device 102 sends an invitation to the client devices 104 and 106via email, SMS, or other communication channels. In steps 1508 and 1510,respectively, the client devices 104 and 106 join the doctor's waitingroom (which may be specific to the doctor or may be a general waitingroom in the medical facility). In step 1512, the server 1204 a notifiesthe control device 102 of the available status of the client devices 104and 106. In steps 1514 and 1516, respectively, the server 1204 anotifies the client devices 104 and 106 of the available status of thecontrol device 102. In steps 1518 and 1520, respectively, the clientdevices 104 and 106 notify the control device 102 that they are in thewaiting room.

Referring specifically to FIG. 15B, in step 1522, the control device 102sends patient education material to the client device 106, which theclient device 106 displays to its user in step 1524. As this process hasbeen described in previous embodiments, it is not further detailed here.In step 1526, the control device 102 starts a session (e.g., an audio orvideo call) with the client device 104. In step 1528, the control device102 notifies the server 1204 a that it is on a call, and the server 1204a updates the control device's status to the client device 106 in step1530. At this time, there is an active call between the control device102 and the client device 104 as indicated by step 1532.

In step 1534, the control device 102 invites the client device 106 tojoin the ongoing call of step 1532. However, the control device 102 doesnot host the conference call itself and so sends a message to the server1204 a that it is joining a conference call. The message may include theidentify of the other participants (e.g., the client devices 104 and106) and may serve as an instruction to the server 1204 a to set up thecall. In steps 1540 and 1542, respectively, the client devices 104 and106 send messages to the server 1204 a to join the conference call. Instep 1542, the server 1204 a connects to a server 1204 b, which is theserver 1204 b of FIGS. 12C and 12D in the present example.

Referring specifically to FIG. 15C, in steps 1544, 1546, and 1548, theserver 1204 b joins the control device 102, client device 104, andclient device 106 in a conference call. As the server 1204 b manages themedia for the conference call, media for the call (e.g., audio and/orvideo) flows between the server 1204 b and each of the control device102, client device 104, and client device 106 as shown by arrows 1550,1552, and 1554, respectively. File sharing data also flows between theserver 1204 b and each of the control device 102, client device 104, andclient device 106 as shown by arrows 1556, 1558, and 1560, respectively.

Referring specifically to FIG. 15D, in step 1562, the control device 102may send patient education material to the server 1204 b, which thensends the material to the client devices 104 and 106 in steps 1564 and1566, respectively. The client 104 displays the material to its user instep 1568 and the client device 106 displays the material to its user instep 1570. Chat messages also flow between the server 1204 b and each ofthe control device 102, client device 104, and client device 106 asshown by arrows 1572, 1574, and 1576, respectively.

Referring specifically to FIG. 15E, in step 1578, the control device 102may initiate screen sharing to share its screen with the client devices104 and 106. In the present example, screensharing uses a third server1204 c, but this server may be combined with the server 1204 b in otherembodiments as shown, for example, in FIG. 15G. In step 1580, the server1204 a notifies the server 1204 b of the screenshare, and the server1204 b (which is handling the conference call) sends a message to theserver 1204 c to set up the screenshare in step 1581. In step 1582, theserver 1204 c sends a message to the server 1204 b with screensharesession information, and the server 1204 b passes this information tothe server 1204 a in step 1583. In step 1584, the server 1204 a sendsthe screenshare information to the control device 102. In step 1585, thecontrol device 102 sends screen share data to the server 1204 c, and theserver 1204 c sends the data to the client devices 104 and 106 in steps1586 and 1587, respectively.

Referring specifically to FIG. 15F, in step 1588, the control device 102instructs the server 1204 b to end the conference call. The server 1204b instructs the server 1204 c to end the screenshare session in step1589 and sends a message to the server 1204 a that the conference callis being ended in step 1590. In steps 1591-1593, respectively, theserver 1204 a notifies the control device 102, the client device 104,and the client device 106 that the conference is over. In step 1594, thecontrol device 102 indicates to the server 1204 a that it is available.In step 1595, the control device 102 instructs the server 1204 a todisconnect the client devices 104 and 106 from the waiting room. Theserver 1204 a notifies the client devices 104 and 106 of thedisconnection in steps 1596 and 1597, respectively.

Referring to FIG. 15G, an alternate embodiment of some steps of FIGS.15E and 15F is illustrated with the servers 1204 b and 1204 c combinedinto a single server 1204 b. Accordingly, FIG. 15G illustrates theinitiation of the screenshare in step 1578 (FIG. 15E) through the endingof the conference call in step 1590 (FIG. 15F) using the single server1204 b. As shown, steps 1581, 1582, and 1589 of FIGS. 15E and 15F areomitted, and other steps (e.g., steps 1585-1587) involve the server 1204b. It is understood that steps similar or identical to steps 1581, 1582,and 1589 may still occur internally within the architecture of theserver 1204 b.

Referring to FIGS. 16A and 16B, one embodiment of a sequence diagramillustrates a portion of an alternate process 1600 by which the controldevice 102 (representing a doctor) within the medical environment ofFIGS. 12A and 12B may communicate with client devices 104 (representinga first patient) and 106 (representing a second patient) in a conferencecall. In the present example, the conference call is initiated directlywith both client devices rather than adding a participant to an activecall as shown with respect to FIGS. 15A-15G. Accordingly, only theinitial steps are shown in FIGS. 16A and 16B, as later steps areidentical to those in FIGS. 15A-15G.

Steps 1602-1624 are identical to steps 1502-1524 of FIGS. 15A and 15Band are not described in detail with respect to the present example. Instep 1626, rather than initiate a call only with the client 104 as wasdone in step 1526 (FIG. 15B), the control device 102 sends a message tothe server 1204 a to begin the conference call with both client devices104 and 106. In steps 1628, 1630, and 1632, respectively, the server1204 a sends conference messages to the control device 102, clientdevice 104, and client device 106. The control device 102, client device104, and client device 106 send messages to the server 1204 a to jointhe conference call in steps 1634, 1636, and 1638, respectively. In step1640, the server 1204 a connects to the server 1204 b (similar to step1542 of FIG. 15B). The process 1600 then proceeds in the same manner asthe process 1500 in steps 1544-1597 of FIGS. 15C-15G.

Referring to FIGS. 17A-17F, one embodiment of a sequence diagramillustrates a process 1700 by which the control device 102 (representinga doctor) within the medical environment of FIGS. 12A and 12B maycommunicate with client devices 104 (representing a first patient) and106 (representing a second patient) in a conference call. The presentexample is similar to the process 1500 of FIGS. 15A-15F with theaddition of recording some or all portions of the conference call. It isunderstood that some steps (such as file sharing or screen sharing) mayoccur at different times than those illustrated in FIGS. 17A-17F (or notat all), and that such steps are included to show how general messageflows for various features may be incorporated. Although not shown, itis understood that the alternate conference call establishment of FIG.15G may be incorporated into FIGS. 17A-17F as described with respect toFIGS. 15A-15F.

Referring specifically to FIG. 17A, in step 1702, the control device 102logs into a server 1204 a, which is the server 1204 a of FIGS. 12C and12D in the present example. In steps 1704 and 1706, respectively, thecontrol device 102 sends an invitation to the client devices 104 and 106via email, SMS, or other communication channels. In steps 1708 and 1710,respectively, the client devices 104 and 106 join the doctor's waitingroom (which may be specific to the doctor or may be a general waitingroom in the medical facility). In step 1712, the server 1204 a notifiesthe control device 102 of the available status of the client devices 104and 106. In steps 1714 and 1716, respectively, the server 1204 anotifies the client devices 104 and 106 of the available status of thecontrol device 102. In steps 1718 and 1720, respectively, the clientdevices 104 and 106 notify the control device 102 that they are in thewaiting room.

Referring specifically to FIG. 17B, in step 1722, the control device 102sends patient education material to the client device 106, which theclient device 106 displays to its user in step 1724. As this process hasbeen described in previous embodiments, it is not further detailed here.In step 1726, the control device 102 starts a session (e.g., an audio orvideo call) with the client device 104. In step 1728, the control device102 notifies the server 1204 a that it is on a call, and the server 1204a updates the control device's status to the client device 106 in step1730. At this time, there is an active call between the control device102 and the client device 104 as indicated by step 1732.

In step 1734, the control device 102 invites the client device 106 tojoin the ongoing call of step 1732. However, the control device 102 doesnot host the conference call itself and so sends a message to the server1204 a that it is joining a conference call. The message may include theidentify of the other participants (e.g., the client devices 104 and106) and may serve as an instruction to the server 1204 a to set up thecall. In steps 1740 and 1742, respectively, the client devices 104 and106 send messages to the server 1204 a to join the conference call. Instep 1742, the server 1204 a connects to a server 1204 b, which is theserver 1204 b of FIGS. 12C and 12D in the present example.

Referring specifically to FIG. 17C, in steps 1744, 1746, and 1748, theserver 1204 b joins the control device 102, client device 104, andclient device 106 in a conference call. As the server 1204 b manages themedia for the conference call, media for the call (e.g., audio and/orvideo) flows between the server 1204 b and each of the control device102, client device 104, and client device 106 as shown by arrows 1750,1752, and 1754, respectively.

In step 1756, the control device 102 notifies the server 1204 a to beginrecording. In step 1758, the server 1204 a instructs the server 1204 b(which may be the same server as the server 1204 a or a differentserver) to setup a recording session for the control device 102 and theclient device 104. At this point, there is an active recording sessionwith the server 1204 b sending conference call data to the server 1204 cfor recording as shown by step 1760.

Referring specifically to FIG. 17D, patient education material may besent and viewed in steps 1762-1767 as described in previous examples. Insteps 1768-1770, chat messages may be exchanged as described in previousexamples. As indicated by arrow 1771, this information may be sent fromthe server 1204 b to the server 1204 c for recording. It is understoodthat the recording may occur in various manners, including the sendingof batched data or via streaming, and so may occur as media, files, chatmessages, and/other data are being received from and sent to the variousdevices 102, 104, and 106. Accordingly, the active recording may be anongoing process and is not limited to the illustrated arrows.

Referring specifically to FIG. 17E, a screenshare session may beestablished in steps 1772-1780 using a fourth server 1204 d (which maybe combined with the server 1204 b) as previously described. Filesharing may occur in steps 1781-1783 as previously described. Asindicated by arrow 1784, this information may be sent from the server1204 b to the server 1204 c for recording.

Referring specifically to FIG. 17F, in step 1785, the control device 102sends a message to the server 1204 b to stop recording. In step 1786,the server 1204 b instructs the server 1204 c to stop recording. In step1787, the control device instructs the server 1204 b to end theconference call. In step 1788, the server 1204 b sends a message to theserver 1204 d to end the screensharing session. In step 1789, the server1204 b notifies the server 1204 a that the conference has ended.

In steps 1790-1792, respectively, the server 1204 a notifies the controldevice 102, the client device 104, and the client device 106 that theconference is over. In step 1793, the control device 102 indicates tothe server 1204 a that it is available. In step 1794, the control device102 instructs the server 1204 a to disconnect the client devices 104 and106 from the waiting room. The server 1204 a notifies the client devices104 and 106 of the disconnection in steps 1795 and 1796, respectively.In step 1797, the control device 102 may retrieve the recording from theserver 1204 b and play back the session in step 1798.

FIGS. 18-26 illustrate embodiments of a GUI that may be used by thecontrol device of FIGS. 1A, 1B, and 12A-12D. Although the particularexamples of FIGS. 18-26 may contain details relevant specifically to themedical environment 1200 of FIGS. 12A and 12B, various illustratedfeatures may be applied to many different environments. It is understoodthat the functionality needed to provide the GUIs may be present on thecontrol device 102 (e.g., as an application) or may be present on aserver that is accessible to the control device, in which case thecontrol device may use an application or a browser (with or withoutclient side downloads such as applets) to access the functionality. Itis understood that the illustrated displays and functionality may varywith more or fewer functions than shown, and placement, size, and otherdetails may also change depending on the particular implementation.

As described in previously incorporated U.S. Provisional ApplicationSer. No. 63/176,419, filed on Apr. 19, 2021, and entitled SYSTEM ANDMETHOD FOR HIGHLY SCALABLE BROWSER-BASED AUDIO/VIDEO CONFERENCING, insome embodiments, audio/video data for the control device and/or clientdevice(s) for single and/or conference calls may be displayed via abrowser (e.g., using Chrome, Safari, Internet Explorer, Brave, Opera, ora similar browser) without the use of a client-side application orplug-in on the device. It is understood that in other embodiments thepresent disclosure may be applied to environments in which a device usesan application or browser plug-in to communicate with the other deviceor MCU 1204 b, and the description of browser only communications is notintended to be limiting.

By relying strictly on the browser's inherent capabilities without theuse of applications or plug-ins, the ability to join and participate inthe conference call is available to any device with a browser. Thissimplifies joining a conference call, and enables joining even if thedevice does not permit the download or installation of applications orbrowser plug-ins. Furthermore, this provides a level of security to thedevice, as there are no downloads to be installed or authorized in orderto access the conference call. In addition, as many browsers are widelyused and frequently updated for security reasons, the user of the deviceneed not be concerned about potential application or plug-in flaws thatmight compromise the device's security if not updated. In addition, byrelying only on the device's browser, there is less chance of needing anupdate before joining a conference call, as might happen if anapplication or a plug-in has not been used for a while. This alsoenables even mobile devices to fully participate in a conference callusing only their built-in browser (or another browser that is selectedby the user).

To accomplish this browser focused conferencing, the conference server1204 b may use the WebRTC framework to provide complete conferencefunctionality. The solution also supports the Unified Plan for SDPs assupported by Safari, in addition to Plan-B that is supported by Chromeand other platforms. Further support may be provided using across-platform JavaScript SDK that is fully featured.

Referring to FIG. 18 , one embodiment of a GUI 1800 illustrates a screendisplay of a control device 102 that enables a user of the device toview a virtual waiting room, select client devices with which tocommunicate, view call logs, and/or perform other operations. In thepresent example, the GUI 1800 provides various functions andinformation. Actuators (e.g., buttons) are present for different views,including a dashboard button 1802, a conference button 1804, a call logsbutton 1806, and a sigh out button 1808. A selector button 1810 enablesthe left panel to be hidden/unhidden.

A waiting room 1812 provides a view of anyone waiting in the virtualwaiting room. In the present example, the waiting room is currentlyempty. An invitation panel 1814 provides a link and a way in which toinvite users to the waiting room or a call. For purposes of example, theinvitation panel 1814 includes a link 1816 that may be sent to invitees,where clicking on the link will place them in the waiting room. Forexample, sending the link result in step 1304 of FIG. 13A. A copy button1818 enables the link to be copied for convenience. An invite button1820 allows the selection of a communication channel for the invitation,which is either email or SMS as shown by a popup window 1822.

Referring to FIG. 19 , another embodiment of the GUI 1800 illustrates asingle user identified as “Tim” now present in the waiting room 1812. Apane 1902 indicates the name and status (e.g., waiting) of the clientand an options selector 1904 may be present.

Referring to FIG. 20 , another embodiment of the GUI 1800 illustrates asecond user identified as “Siva” now present in the waiting room 1812. Apane 2002 indicates the name and status (e.g., waiting) of the clientand an options selector 2004 may be present.

Referring to FIG. 21 , another embodiment of the GUI 1800 illustrates awindow 2102 that appears when the selector 1904 is actuated. In thepresent example, the window 2102 is a popup window, but it is understoodthat other view options may be provided. The window 2102 may include apicture 2104 of the client along with their name and status. A statusindicator 2106 may be provided to visually indicate their status usingcolor and/or other representations. For example, a green dot mayindicate ready and waiting, a yellow dot may indicate the client hasplaced the call on hold, has a poor connection, or is not ready (e.g.,an appointment time has not been reached), and a red dot may indicatelack of a connection.

Various buttons or other actuators may provide functionality, includinga chat button 2108 to initiate a chat message, a file share button 2110to initiate a file share, an audio call button 2112 to initiate an audiocall, and a video call button 2114 to initiate a video call. A patienteducation button 2116 may be used to allow the selection of patienteducation material to send to the client. A disconnect button 2118enables disconnection of the current client, which may disconnect themfrom the waiting room.

Referring to FIGS. 22A-22C, another embodiment of the GUI 1800illustrates a window 2202 that appears when the selector 2116 isactuated. Referring specifically to FIG. 22A, in the present example,the window 2202 includes patient education material in the form ofvideos that may be selected and sent to a user (in this came, the userTim). It is understood that the materials may be presented and/orselected in different ways, including checkboxes, radio buttons, and/ordrop down lists. Furthermore, it may be possible to select multiplevideos. A close button 2206 enables the window 2202 to be closed, and aplay button 2208 enables the selected content to be played (e.g., sentto the user for display). FIG. 22B illustrates an example where thepatient education material is in the form of text/images. FIG. 22Cillustrates an example where the patient education material is in theform of both video and text/images.

Referring to FIG. 23 , one embodiment of the GUI 1800 illustrates avideo call display for the control device 102 when a video call isestablished with user Tim. A photo button 2302 enables a photo to becaptured by the control device 102. A conference call button 2304enables one or more participants to be added to the conference call. Ascreenshare button 2306 enables the user of the control device 102 toshare the screen (e.g., step 1366 of FIG. 13D). A file transfer (orshare) button 2308 enables a file to be sent (e.g., step 1344 of FIG.13C).

A waiting room view 2310 provides a view of users in the virtual waitingroom, with Siva currently waiting. An options selector 2312 may provideoptions (e.g., some or all of the options of the window 2102 of FIG. 21) for interacting with Siva. A view window 2314 provides a self-view(e.g., a view using a camera of the control device 102). A view window2316 provides a view of Tim (e.g., using the camera of Tim's clientdevice). A selector button 2318 enables the left panel to behidden/unhidden.

Various buttons may be used to provide additional functions and/orcontrol the call. A chat button 2320 may be provided to enable textmessaging. A video button 2322 may be provided to enable and disablevideo. A pause button 2324 may be used to place the call on hold, andmay be replaced by a play button (not shown) to remove the call fromhold. A mic button 2326 may be used to mute and unmute audio. Adisconnect button 2328 may be used to end the call.

Referring to FIG. 24 , one embodiment of the GUI 1800 illustrates awindow 2402 that enables a conference call to be established. In thepresent example, the window 2302 enables one or both of the users in thewaiting room to be invited to the call.

Referring to FIG. 25 , one embodiment of the GUI 1800 illustrates aconference call display for the control device 102 when a conferencecall is established with two users Siva and Tim. A participants button2502 enables a participant view. An add participants button 2504 enablesone or more participants to be added to the conference call. Ascreenshare button 2506 enables the user of the control device 102 toshare the screen. A file share button 2508 enables a file to be sent.

Referring to FIG. 26 , one embodiment of the GUI 1800 illustrates calllogs that are displayed after the call logs button 1806 is selected. Asshown, call logs may be selected, sorted, and viewed based on a numberof criteria.

Referring to FIG. 27 , one embodiment of a GUI 2700 illustrates a screendisplay of a client device (e.g., one of the client devices 104, 106,and 108) that enables a user of the device to join a virtual waitingroom and participate in a call via the waiting room. For purposes ofexample, the client device 104 is a smart phone and the display uses abrowser with controls 2702, but other devices and/or applications may beused. The GUI 2700 may include a viewing portion 2704 showing a web pagethat provides the user with the ability to join the waiting room byentering a name in a name field 2706 and selecting a button 2708.

Referring to FIG. 28 , one embodiment of the GUI 2700 shows the window2704 after check in. A status indicator 2802 for the doctor may visuallyindicate availability. A window 2804 may display patient educationmaterial (e.g., such as that sent in step 1336 of FIG. 13B). It isunderstood that controls for the material's display may be provided andmay vary based on the type of material (e.g., text or video).

Referring to FIG. 29 , one embodiment of the GUI 2700 shows the window2704 while waiting, but without patient education material.

Referring to FIG. 30 , one embodiment of a computer system 3000 isillustrated. The computer system 3000 is one possible example of asystem component or computing device such as a communication device, adocument server, an endpoint, and/or an access server. The computersystem 3000 may include a controller (e.g., a central processing unit(“CPU”)) 3002, a memory unit 3004, an input/output (“I/O”) device 3006,and a network interface 3008. The components 3002, 3004, 3006, and 3008are interconnected by a transport system (e.g., a bus) 3010. A powersupply (PS) 3012 may provide power to components of the computer system3000, such as the CPU 3002 and memory unit 3004. It is understood thatthe computer system 3000 may be differently configured and that each ofthe listed components may actually represent several differentcomponents. For example, the CPU 3002 may actually represent amulti-processor or a distributed processing system; the memory unit 3004may include different levels of cache memory, main memory, hard disks,and remote storage locations; the I/O device 3006 may include monitors,keyboards, and the like; and the network interface 3008 may include oneor more network cards providing one or more wired and/or wirelessconnections to a network. Therefore, a wide range of flexibility isanticipated in the configuration of the computer system 3000.

The computer system 3000 may use any operating system (or multipleoperating systems), including various versions of operating systemsprovided by Microsoft (such as WINDOWS), Apple (such as Mac OS X), UNIX,and LINUX, and may include operating systems specifically developed forhandheld devices, personal computers, and servers depending on the useof the computer system 3000. The operating system, as well as otherinstructions (e.g., for the processes and message sequences describedherein), may be stored in the memory unit 3004 and executed by theprocessor 3002. For example, if the computer system 3000 is the controldevice 102 or a client device 104, 106, 108, the memory unit 3004 mayinclude instructions for performing some or all of the message sequencesand methods described with respect to such devices in the presentdisclosure.

The network 3016 may be a single network or may represent multiplenetworks, including networks of different types. For example, thecontrol device 102 or a client device 104, 106, 108 may be coupled to anetwork that includes a cellular link coupled to a data packet network,or data packet link such as a wide local area network (WLAN) coupled toa data packet network. Accordingly, many different network types andconfigurations may be used to establish communications between thecontrol device 102, client devices 104, 106, 108, servers, and/or othercomponents described herein.

Exemplary network, system, and connection types include the internet,WiMax, local area networks (LANs) (e.g., IEEE 802.11a and 802.11g wi-finetworks), digital audio broadcasting systems (e.g., HD Radio, T-DMB andISDB-TSB), terrestrial digital television systems (e.g., DVB-T, DVB-H,T-DMB and ISDB-T), WiMax wireless metropolitan area networks (MANs)(e.g., IEEE 802.16 networks), Mobile Broadband Wireless Access (MBWA)networks (e.g., IEEE 802.20 networks), Ultra Mobile Broadband (UMB)systems, Flash-OFDM cellular systems, and Ultra wideband (UWB) systems.Furthermore, the present disclosure may be used with communicationssystems such as Global System for Mobile communications (GSM) and/orcode division multiple access (CDMA) communications systems. Connectionsto such networks may be wireless or may use a line (e.g., digitalsubscriber lines (DSL), cable lines, and fiber optic lines).

Communication among the control device 102, client devices 104, 106,108, servers, and/or other components described herein may beaccomplished using predefined and publicly available (i.e.,non-proprietary) communication standards or protocols (e.g., thosedefined by the Internet Engineering Task Force (IETF) or theInternational Telecommunications Union-Telecommunications StandardSector (ITU-T)), and/or proprietary protocols. For example, signalingcommunications (e.g., session setup, management, and teardown) may use aprotocol such as the Session Initiation Protocol (SIP), while datatraffic may be communicated using a protocol such as the Real-timeTransport Protocol (RTP), File Transfer Protocol (FTP), and/orHyper-Text Transfer Protocol (HTTP). A sharing session and othercommunications as described herein may be connection-based (e.g., usinga protocol such as the transmission control protocol/internet protocol(TCP/IP)) or connection-less (e.g., using a protocol such as the userdatagram protocol (UDP)). It is understood that various types ofcommunications may occur simultaneously, including, but not limited to,voice calls, instant messages, audio and video, emails, documentsharing, and any other type of resource transfer, where a resourcerepresents any digital data.

While the preceding description shows and describes one or moreembodiments, it will be understood by those skilled in the art thatvarious changes in form and detail may be made therein without departingfrom the spirit and scope of the present disclosure. For example,various steps illustrated within a particular sequence diagram or flowchart may be combined or further divided. In addition, steps describedin one diagram or flow chart may be incorporated into another diagram orflow chart. Furthermore, the described functionality may be provided byhardware and/or software, and may be distributed or combined into asingle platform. Additionally, functionality described in a particularexample may be achieved in a manner different than that illustrated, butis still encompassed within the present disclosure. Therefore, theclaims should be interpreted in a broad manner, consistent with thepresent disclosure.

What is claimed is:
 1. A method for managing multiple independentcommunication sessions between a control device and at least first andsecond client devices, the method comprising: establishing, between acontrol device and a first client device, a first communication session;establishing, between the control device and a second client device, asecond communication session; participating, by the control device, inthe first communication session while the second communication sessionis on hold; placing, by the control device, the first communicationsession on hold; injecting, by the control device, a first visual mediafile into the first communication session, wherein the first visualmedia file is to be displayed on the first client device while the firstcommunication session is on hold; and switching, by the control device,to the second communication session while the first communicationsession is on hold.
 2. The method of claim 1 wherein injecting the firstvisual media file into the first communication session furthercomprises: receiving, by the control device, a selection of the firstvisual media file from a plurality of available visual media files; andsending the first visual media file to the first client device, whereinthe entire first visual media file is received before being displayed.3. The method of claim 1 wherein injecting the first visual media fileinto the first communication session further comprises: receiving, bythe control device, a selection of the first visual media file from aplurality of available visual media files; and streaming the firstvisual media file to the first client device while the firstcommunication session is on hold.
 4. The method of claim 1 whereininjecting the first visual media file into the first communicationsession further comprises: receiving, by the control device, a uniformresource locator (URL), wherein the URL represents a location of thefirst visual media file on a server accessible to the first clientdevice; and sending the URL to the first client device.
 5. The method ofclaim 1 wherein the first and second client devices are invited tocommunicate with the control device via an identical uniform resourcelocator (URL).
 6. The method of claim 1 further comprising: receiving,by the control device, a request for a third communication session froma third client device; and adding, by the control device, the thirdclient device to a virtual waiting room, wherein the third communicationsession is automatically placed on hold when added to the virtualwaiting room.
 7. The method of claim 1 wherein the first client deviceis able to participate in the first communication session using only astandard browser present on the first client device, wherein no plug-inor other application download is required to participate in the firstcommunication session.
 8. A method for injecting a visual media fileinto a communication session between a control device and a clientdevice, the method comprising: establishing, between a control deviceand a client device, a communication session; placing, by the controldevice, the communication session on hold; injecting, by the controldevice, a visual media file into the communication session, wherein thevisual media file is to be displayed on the client device while thecommunication session is on hold; and removing, by the control device,the hold to continue the communication session.
 9. The method of claim 8wherein injecting the visual media file into the communication sessionfurther comprises: receiving, by the control device, a selection of thevisual media file from a plurality of available media files; and sendingthe visual media file to the client device, wherein the entire visualmedia file is received before being displayed.
 10. The method of claim 8wherein injecting the visual media file into the communication sessionfurther comprises: receiving, by the control device, a selection of thevisual media file from a plurality of available media files; andstreaming the visual media file to the client device while thecommunication session is on hold.
 11. The method of claim 8 whereininjecting the visual media file into the communication session furthercomprises: receiving, by the control device, a uniform resource locator(URL), wherein the URL represents a location of the visual media file ona server accessible to the client device; and sending the URL to theclient device.
 12. The method of claim 8 further comprising injecting,by the control device, a second visual media file into the communicationsession while the communication session is not on hold, wherein thesecond visual media file is to be played on the client device while thecommunication session is not on hold.